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I.  INTRODUCTION 


1.1  Program  Objective 


The  main  objective  of  this  program  is  to  construct  a 
flexible  testbed  -for  the  evaluation  of  the  effectiveness  of 
computer-based  expert  systems  in  military  training  and 
planning.  \ 


-  The  technical  approach  consists  of  simulating  the 

characteristics  of  expert  systems  in  a.  game-like  environment. 

Such  characteristics  include  (1)  friendly  system  user  inter¬ 
act  on  ?(2?syste;n  explanations  of  rationale  about  decision 
recommendations /T3?an  ability  to  make  relevant  suggestions 
and  comments  about  situatiorwissessments  and  about  plans 
proposed  by  the  user,  and  (4)' the  use  of  high-level  strategic 
concepts  and  terminology. 

The  required  software  for  such  a  program  includes  (1)  a 
game  environment  simulator  called  the  Scenario  Generator , 
042^-a  simulated  expert  system  for  the  game  called  the  ^Expert 
Ai  an  Optimality  Algorithm  for  computing  the  best  decisions 

in  any  situation,  and  (4)  an  Evaluation  Module  for  recording 
execution  histories  and  performance  parameters.  The  expert 
system  will  monitor  the  progress  of  the  game  and  can  be  inter¬ 
rogated  as  the  user  sees  fit.  A  facility  will  also  be  provided 
for  evaluating  tne  user's  performance  under  different  modes  of 
consultation  with  the  expert  system.. 
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1 .3  Progress 


To  date,  the  following  progress  has  been  made  on  the 
design  and  implementation  of  the  expert  system. 

(1)  Design  of  the  Scenario  Generator. 

(2)  Prototype  implementation  of  the  Scenario  Generator  game 
for  testing  purposes. 

(3)  Completion  of  the  Optimality  Algorithm  Design. 

(4)  Implementation  and  execution  of  Optimality  Algorithm 
resulting  in  a  set  of  optimal  decision  tables. 

(5)  Design  of  the  Expert  System  and  integration  into  the 
existing  Scenario  Generator. 

(6)  Design  of  the  Experimental  Plan. 

Plans  for  project  continuation  include  the  following  major  tasks. 

(1)  Implementation  of  the  full  expert  system  including 
system  explanations  of  recommendations. 

(2)  Implementation  and  integration  of  the  Evaluation  Module 
for  measuring  user  performance. 

(3)  Completion  of  the  evaluation  experiments. 
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II.  THEORETICAL  FOUNDATIONS 


2.1  Expert  Systems  with  Ground  Truth 

In  a  problem  environment  involving  multiple  decision  points, 
"ground  truth"  is  a  complete  description  of  the  optimal 
strategy  for  all  possible  situations  the  decision  maker  may 
encounter.  These  optimal  decisions  must  be  identifiable  by 
computation  and  must  not  be  dependent  upon  the  judgment  of 
humans  regardless  of  their  expertise.  An  expert  system,  armed 
with  ground  truth,  could  make  perfect  recommendations  to  the 
decision  maker  at  all  times.  If  the  environment  contains 
uncertain  events,  the  ground  truth  can  be  accurate  only  up 
to  the  highest  expected  utility.  That  is,  ground  truth  may 
not  guarantee  successful  outcomes;  it  just  means  that  no  other 
decision  strategy  could  produce  better  performance  in  the  long 
run. 

Ground  truth,  then,  is  a  property  of  the  environment  and  not 
of  the  expert  system.  Most  expert  systems  do  not  have  built-in 
ground  truth.  Their  recommendations  are  bafed  on  human  knowledge 
and  experience  encoded  as  rules.  The  reason  for  this  is  obvious. 
Any  environment  complex  enough  to  warrent  the  construction  of 
an  expert  system  is  usually  too  complex  to  compute  ground  truth. 
However,  the  existence  of  ground  truth  permits  the  calibration 
of  performance  measures  in  expert  system  evaluation.  Learning 
curves  for  new  users  could  be  generated  by  comparing  decision 
performance  with  and  without  the  recommendations  of  the  expert 
system,  but  the  results  would  be  unreliable  since  the  user's 
expertise  continuously  increases  simply  from  contact  with  the 
environment.  On  the  other  hand,  ground  truth  provides  an  upper 
bound  on  performance  and  a  standard  against  which  performance  can 
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be  measured.  In  an  environment  containing  uncertain  events, 
evaluation  without  ground  truth  must  rely  on  the  outcomes 
of  the  decisions  rather  than  on  the  quality  of  the  decisions 
themselves.  The  results  would  then  be  clouded  by  the  random 
processes  within  the  environment.  With  ground  truth  as  a 
standard,  precise  measurements  of  decision  performance  can 
be  attained  regardless  of  the  outcomes  of  the  experiments. 


2.2  Scenario  Generator  Requirements 

The  task,  then,  is  to  devise  a  decision  environment  which  is 
complex  enough  to  be  challenging  to  a  decision  maker  and,  yet, 
simple  enough  to  permit  the  calculation  of  ground  truth.  These 
objectives  lead  to  the  following  requirements  for  the  expert 
system  testbed  Scenario  Generator. 

(1)  Environment  complex  enough  to  be  challenging  to  a 
decision  maker. 

(2)  Environment  simple  enough  to  calculate  ground  truth. 

(3)  Environment  amenable  to  the  construction  of  and 
use  of  an  expert  system  for  decision  recoimnenuati ons 

{4)  Environment  containing  uncertain  events. 

(5)  Environment  containing  information  feedback  during 
the  decision  process. 

(6)  Environment  within  the  military  domain  and  which 
embodies  a  military  decision  problem. 
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Since  uncertain  events  and  information  feedback  are  common 
elements  of  decision  environments,  their  inclusion  in  the 
Scenario  Generator  is  important  for  the  transfer  of  evaluation 
results  to  real-world  expert  systems. 

The  selected  environment  that  satisfies  the  above  criteria 
is  a  military  decision  game.  An  expert  system  designed  around 
a  game  would  provide  recommendations  about  the  best  "moves" 
in  different  situations.  Ground  truth  can  be  established  (if 
the  game  is  not  too  complex)  since  the  criteria  for  winning  and 
losing  are  specified  precisely  in  the  rules.  The  only  disadvantage 
is  that  games  are  not  generally  very  realistic.  However,  as  will 
be  shown  in  the  next  section,  a  methodology  can  be  described 
which  will  permit  the  construction  of  a  ground  truth  expert 
system  in  any  environment.  The  experimental  results  will  then 
be  transferrable. 


2.3  The  General  Model 


The  scenario  game  environment  is  founded  upon  a  general  model  for 
a  class  of  ground  truth  expert  systems.  The  major  components  of 
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(1)  Well-defined  decision  points  with  all  choice  options 
specified  in  advance. 

(2)  Well-defined  event  points  with  all  outcomes  and 
associated  probabilities  specified. 
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(3)  Well-defined  information  feedback  points  with  all 
possible  combinations  of  "reports"  specified. 

Figure  1  shows  notational  symbols  for  each  of  the  components: 
decision  points,  event  points,  and  report  points.  An  "identi¬ 
fication  point"  is  a  place  during  the  decision  process  where 
an  important  observation  is  made  when  it  does  not  coincide  with 
one  of  the  other  components.  By  connecting  symbols  together  to 
form  a  tree,  a  class  of  decision  environments  can  be  constructed 
which  leads  directly  to  the  design  of  the  ground  truth  expert 
system. 


A  complete  decision  structure  should  have  the  following  charac¬ 
teristics. 

(1)  Start  with  a  decision  point. 

(2)  Possess  normalized  probabilities  on  each  branch  leading 
out  from  an  event  point. 

(3) Posses$  utility  values  on  every  termination  branch. 


Once  built,  the  decision  structure  dictates  the  procedure  for 
calculating  the  ground  truth.  At  every  decisiun  point,  the  value 
of  the  decision  (for  all  possible  situations)  is  equal  to  the 
maximum  of  the  values  on  its  branches.  At  every  event  point,  the 
value  of  the  event  is  equal  to  the  expected  value  of  the  utilities 
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utility  of  the  information  is  equal  to  the  utility  of  its  branch 
given  the  probability  of  the  report.  Thus,  decision  utilities 
are  calculated  by  a  "rollback"  procedure  that  begins  at  the  terminal 
branches  and  ends  at  the  initial  decision  point.  These  calculations 
must  be  performed  for  every  combination  of  decision  situation  and 
every  combination  of  report  outcome.  In  other  words,  a  single 
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tree  diagram  shows  the  sequential  decisions  that  must  be  made, 
but  the  decision  maker  could  be  faced  with  the  sequence  over 
and  over  again  in  a  repetitive  environment  in  which  situational 
parameters  (such  as  resources,  etc.)  are  different  each  time. 
This  leads  to  a  complex  series  of  matrices  necessary  to  rollback 
the  utility  values  and  calculate  the  optimal  decisions  at  every 
decision  point.  The  "Military  Breakthrough"  game,  designed  as 
an  experimental  testbed,  satisfies  the  requirements  of  the 
general  model  and  illustrates  how  these  utility  calculations 
are  made  and  how  they  lead  to  the.. construction  of  the  corres¬ 
ponding  expert  system. 


III.  THE  EXPERT  SYSTEM 


.3.1  Optimal  Decision  Tables 

The  140  decision  tables  provide  the  optimal  decisions  for  any 
situation  encountered  in  the  simulation  game.  Appendix  B 
contains  a  complete  listing  of  all  tables  arranged  on  28  pages: 
one  page  for  each  combination  of  enemy  line  (1-4)  and  enemy 
encampment  (1-7).  Five  tables  are  associated  with  each  situation. 

Table  1:  Attack/Continue/Shallow/Deep 
Table  2:  Attack/Continue  given  Shallow,  not  detected 
Table  3:  Attack/Continue  given  Shallow,  detected 
Table  4:  Attack/Continue  given  Deep,  not  detected 
Table  5:  Attack/Continue  given  Deep,  detected 

Table  1  is  used  when  the  player  encounters  a  new  enemy  encampment 
and  is  faced  with  the  decision  of  whether  or  not  to  send  out 
reconnaissance  (See  Figure  2).  If  reconnaissance  is  chosen,  a 
secondary  decision  is  required  to  determine  the  type:  shallow  (S) 
or  deep  (D).  However,  if  the  player  decides  not  to  send  recon¬ 
naissance,  he  is  left  with  the  decision  to  immediately  attack  (A) 
or  continue  (C)  to  the  next  energy  position.  The  optimal  decisions 
for  these  4  choices  appear  in  Table  1. 

Tha  arfnal  nn+ima  1  Horicinn  rianon/4t  nnnn  *3  f artn r*c  •  /  1  ^  f ha  rnrrnnf 

■  IIV  uv  vwu  i  v|/  v  >>mw  ■  wyv  i  w  i  vm  uf/wn  i  uv  *»V  \*/  vl  1  w  wmi  ■  v"  w 

position,  (2)  the  current  strength  of  the  friendly  forces,  and 
(3)  the  first  report  (XI)  receivad  by  the  player  on  the  status  of 
the  strength  of  the  enemy  forces.  Thus,  decision  Table  1  on  each 
of  the  28  pages  in  the  listing  in  the  Appendix  is  12x64  in  size, 
one  row  for  each  of  the  12  possible  friendly  force  combinations 
and  one  column  for  each  of  the  64  possible  XI  reports.  The  last  line, 
showing  a  friendly  force  of  zero,  is  omitted  since  the  game  would  be 
over  and  no  further  decisions  could  be  made. 
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For  example,  if  the  player  is  facing  enemy  line  #1  and  enemy 
encampment  #3  with  a  force  of  MHH  (meduim  strength  artillery, 
high  strength  infantry,  and  high  strength  tanks)  and  observes 
that  the  report  for  the  enemy  strength  is  NNL  (nil,  nil,  low 
reading  from  top-to-bottom) ,  the  optimal  decision  would  be  "D" , 

("take  Deep  reconnaissance"). 

Tables  2  and  3  are  used  only  when  the  player  has  chosen  shallow 
(S)  reconnaissance  for  the  Initial  decision  (Table  1).  Table  2 
is  used  if  the  reconnaissance  force  is  not  detected  and  Table  3 
is  used  if  the  force  is  detected.  For  these  two  tables,  only  the 
decisions  of  attacking  (A)  or  continuing  (C)  are  relevant.  The 
.  ’column  labels  at  the  top  of  the  page  still  apply,  but  now  represent 
the  outcome  of  report  X2,  a  more  accurate  portrayal  of  the  true 
enemy  strength.  Some  of  the  entries  in  Tables  2  ar.d  3  are  missing 
since  these  particular  combinations  are  impossible  for  report  X2. 

Tables  4  and  $  at  the  bottom  of  each  page  are  used  if  deep  (D) 
reconnaissance  is  made  for  the  initial  decision  in  Table  1.  Again, 
two  tables  are  required.  Table  4  for  "not  detected"  and  Table  5 
for  "detected".  In  these  cases,  however,  only  4  columns  are  necessary 
since  the  true  enemy  strength  is  known  to  the  player  at  this  time 
(as  the  result  of  report  X3)  and  can  be  only  1  of  4  possible  com- 

bl  Haul OH5 . 


Thus,  as  can  be  seen  in  Figure  3,  S  and  D  are  intermediate  decision 
points  whereas  A  and  C  are  final  decision  points.  Numbers  inside 
circles  refer  to  the  Tables  used. 
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Figure  3.  Player  Decisions 
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3.2  Expert  Rule  Formation 


An  expert  system,  which  had  access  .to  all  140  decision  tables, 
could  give  optimal  recommendations  for  every  encountered 
situation.  However,  such  Information  may  not  be  the  most 
valuable  to  a  user.  One  of  the  purposes  of  an  expert  system 
is  to  teach  the  human  user  enough  about  the  characteristics  of 
the  application  environment  so  that  he  acquires  some  expertise. 

As  they  now  exist,  the  decision  tables  are  far  too  complex  to 
learn,  especially  by  point  recommendations  as  individual  situ¬ 
ations  are  encountered  during  simulation  exercises. 

A  more  useful  set  of  recommendations  would  be  based  on  general 
rules  that  capture  most,  but  not  necessarily  all  of  the  optimal 
decisions.  By  observing  the  optimal  tables  listed  in  Appendix  B, 
it  is  clear  that  most  of  the  decisions  fall  into  clusters.  Thus, 
an  expert  system  "rule'1  can  be  defined  as  a  single  recommended 
decision  which  represents  the  majority  of  optimal  decisions 
within  a  specified  region  of  the  decision  space. 

The  accuracy  of  the  expert  system  will  ultimately  depend  upon 
the  uniformity  of  decisions  within  selected  clusters.  The  accuracy 
can,  thus,  he  calculated  precisely.  It  Is  simply  the  total  per¬ 
centage  of  correctly  classified  decision  points.  The  clusters 
should  be  chosen  in  such  a  way  to  produce  high  accuracy  while 
reducing  the  cognitive  complexity  associated  with  learning  the 
rules. 
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Figure  4.  Decision  Table  Clusters 


3.3  Expert  Rales  for  the  Simulation  Game 


The  decisions  for  a  single  table  are  composed  of  a  n.-nber  of 
connected  rules,  one  rule  for  each  rectangular  region.  Figure 
5  shows  the  divisions  for  encampment  #3  in  enemy  line  if 3. 

Table  1  has  3  distinct  rectangular  clusters  and  1  non-rectan- 
gular  region  covering  the  remainder  of  the  table.  To  specify 
the  rules,  only  the  bounderies  for  the  columns  and  rows  are 
necessary.  Thus,  the  upper  left-hand  cluster  (with  mostly  A's) 
falls  between  0  and  2  in  enen\y  strength  and  between  7  and  9  in 
friendly  strength.  A  set  of  3  rules  would  be  required  to  completely 
describe  Table  1.  The  following  form  shows  this  specification 
using  "E"  for  the  enemy  strength  and  "F"  for  the  friendly  strength. 


RULE  # 

*  E 

_£ 

s _ 

F  s 

DO 

ELSE 

1 

0 

2 

7 
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2 

U 

3 

b 

5 

A 

3 

3 

0 

9 

0 

4 

S 

D 

The  above 

description 

can 

be  read  as 

follows: 

(1)  If  the  enemy  strength  is  between  0  and  2  and  the  friendly 
strength  is  between  7  and  9,  attack  (A),  otherwise  follow  Rule  2. 

(2)  If  the  enemy  strength  is  between  0  and  3  and  the  friendly 
strength  is  equal  to  5;  attacks  otherwise  follow  Rule  3. 

(3)  If  the  enemy  strength  is  between  0  and  9  and  the  friendly 
strength  is  between  0  and  4,  send  shallow  reconnaissance  (S), 
otherwise  send  deep  reconnaissance  (D). 


The  last  decision  recommendation  in  the  last  rule  (D)  covers  the 
non-rectangular  portion  of  the  table. 
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Figure  5.  Example  Clustering 


Obviously,  the  accuracy  of  the  resulting  rules  depends  on  the 
accuracy  of  the  individual  regions.  For  the  example  table 
described  above,  the  accuracy  of  the  rules  is  shown  in  the 
following  chart. 


Region 

Correct 

Total 

%  Correct 

Decision 

1 

41 

50 

82% 

A 

2 

36 

40 

90% 

A 

3 

79 

128 

62% 

S 

4 

484 

506 

96% 

D 

Total 

640 

704 

91% 

Thus,  the  expert  rules  for  this  particular  decision  table  are 
91%  correct. 

The  figures  on  the  following  pages  list  all  115  rules  contained 
in  the  current  preliminary  version  of  the  Expert  System.  Figure  6 
cataloges  the  5  main  tables  for  each  of  the  28  positions  and 
their  associated  first  rule  numbers.  For  example,  if  the  player 
is  facing  enemy  line  #3  and  encampment  #1  and  is  considering  a 
decision  about  reconnaissance  (Table  1),  the  applicable  rule 
is  #68.  - 

Figure  7  contains  the  actual  rules.  If  both  enemy  strength 
conditions  (E)  and  both  friendly  strength  conditions  (F)  are 
true,  the  decision  under  the  "DQ“  column  should  be  followed  where 
"A"  means  attack,  "S"  means  take  shallow  reconnaissance,  "D"  means 
take  deep  reconnaissance,  and  "C"  means  continue  (no  attack). 

If  any  of  the  4  conditions  fail,  the  instructions  under  the  "ELSE" 
column  should  be  followed.  These  instructions  are  either  (1)  a 
decision:  A,  S,  D,  or  C,  or  (2)  a  reference  number  which  indicates 
the  next  rule  to  follow. 
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Figure  7.  Expert  System  Game  Rules 
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Figure  7.  (Continued) 


IV.  EXPERIMENTAL  PLAN 


4.1  Objective 

The  objective  of  the  experimental  plan  is  to  formulate  a 
controlled  experimental  environment  and  procedure  for 
assessing  the  expert  system  as  a  training  aid  and  as  a 
performance  aid.  The  experiments  are  designed  to  verify 
a  number  of  hypotheses  about  the  usefulness  of  the  expert 
system  in  various  modes  of  operation.  Human  subjects  will 
be  enlisted  to  participate  in  the  experiments  and  an 
automatic  performance  measurement  module  will  be  installed 
into  the  expert  system  to  monitor  the  decisions  made  by 
the  subjects.  The  following  sections  describe  the  experimental 
plan  in  more  detail. 


4.2  Verifiable  Hypotheses 

Four  verifiable  hypotheses  have  been  selected  for  experimentation. 


T 


H 


3* 


4* 


Performance  improves  with  increased  level  of  aiding. 
As  task  familiarity  increases,  the  usefulness  of  the 
expert  system  as  a  training  aid  increases. 

When. expertise  is  high,  the  benefit  of  the  expert 

A  1 1**  ^  (Am  a  a  4  *4  i  T  ai.ia  w% 

dYdbciii  uo  a  pci  iui  iiianuc  a  i  u  id  8  unci  • 

The  maximum  level  of  expert'  ?  is  reached  sooner 
when  using  the  expert  system. 


Figure  8  shows  the  hypothesized  relationship  between  the  user’s 
learning  curve  (dotted  line)  and  the  benefit  of  an  expert  system 
(solid  line).  When  the  user  is  new  to  the  environment,  aiding 
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Figure  8.  Potential  Benefit  of  Expert  System 
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of  a  particular  subject.  The  subject  simply  plays  one  or  more 
games  with  no  aiding  at  all  and  his  expertise  level  is  the 
average  number  of  correct  decisions ‘made  divided  by  the  total 
number  decisions  made. 

correct  decisions 
expertise  =  total  decisions 


A  second  measure  of  expertise  is  based  on  the  actual  outcomes 
of  the  decision.  This  measure  is  simply  the  subject's  average 
game  score  over  a  number  of  successive  trials. 

expertise  =  average  game  score 

"Task  Familiarity"  can  be  measured  by  combining  the  number  of 
games  played  with  the  total  amount  of  time  interacting  with 
the  game  environment.  The  "Maximum  Level  of  Expertise"  is 
reached  when  the  increase  in  expertise  from  successive  trials 
approaches  zero.  • 

task  familiarity  =  f(#  games,  total  time) 


max  expertise  reached  when: 

expertise..  -  expertise^  <  6 


The  amount  of  aiding  chosen  is  simply  the  number  of  times  the 
subject  used  the  expert  system  divided  by  the  number  of  times 
he/she  could  have  used  it  (in  those  experiments  in  which  there 
is  a  choice). 


amount  of  aiding  = 


#  times  E.S.  used _ 

#  times  E.S.  available 
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4.4  Performance  Measures 


Measures  of  level  of  performance  correspond  to  or  are  functions 
of  the  measured  test  factors.  These  can  be  simple  or  aggregated 
measures. 

(1)  Decision  time  as  a  function  of  expertise 

(2)  Decision  time  as  a  function  of  task  familiarity 

(3)  Learning  rate  (increase  in  task  familiarity  over  time) 

(4)  Mastery  rate  (increase  ir<  expertise  over  time) 

(5)  Tima  to  reach  maximum  level  of  expertise 

(6)  Increased  game  score  over  time 

(7)  Expertise  vs.  amount  of  aiding  used 


4.5  Test.  Design 

The.  test  design  specifies  those  factors  that  will  be  the  objects 
of  the  test  and  those  factors  that  will  be  eliminated  in  order 
to  remove  their  influence  on  the  results.  - 

For  hypothesis  H1  (See  Section  4.2),  the  effects  of  factor  1 
on  factor  3  (See  Section  4.3)  will  be  measured.  Factors  2,  4, 

5,  and  6  will  be  eliminated.  Three  levels  of  expert  system  aiding 
will  be  emp1 oyed :  full,  partial,  and  none.  With  full  aiding,  the 
subjects  will  be  able  to  consult  the  expert  system  any  time 
desired.  With  partial  aiding,  only  selected  decisions  will 
pc.Tfiit  iding.  For  experiments  concerning  ,  a  Latin  Square- 
type  design  Hill  be  used. 
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For  hypotheses  anc*  H^,  the  effects  of  factors  1  and  6  will 
be  assessed  on  factor  A3.  This  means  that  the  rate  of  change 
in  level  of  expertise  will  be  measured.  Factors  2,  4,  and  5 
will  be  -vitninated.  Repeating  single  factor  experiments  will 
be  used. 

For  hypothesis  H^,  the  effects  of  factor  1  on  factor  4  will  be 
measured.  Factors  2,  3,  5,  and  6  will  be  eliminated.  Again, 
single  factor  experiments  with  repetitions  will  be  used. 

4 . 6  Experimental  Procedure 

Every  experiment  will  be  carried  out  in  4  phases:  (1)  definition, 
(2)  design,  (3)  data  collection,  and  (4)  data  analysis.  The 
definition  phase  will  concern  the  precise  plan  for  carrying 
out  each  experiment.  This  includes  the  dependent  and  independent 
variables  (factors),  the  number  of  observations  taken,  the  number 
of  subjects,  the  number  of  experimental  groups,  etc.  The  design 
phase  concerns  the  type  of  test  performed:  Latin  Square,  repeated 
single  factor  experiments,  etc.  The  data  collection  phase  will 
occur  during  the  experiments  and  will  be  automated  due  to  the 
implementation  of  the  Performance  Evaluation  Module  within  the 
expert  system.  Finally,  data  analysis  will  verify  or  reject  the 
proposed  hypotheses. 

Since  continued  use  of  the  expert  system  during  experimentation 
increases  task  familiarity  and  expertise,  the  experimentation 
order  influences  the  results.  Thus,  a  Latin  Square  Design  is 
appropriate  for  eliminating  this  effect.  A  Latin  Square  is  a 
matrix  that  specifies  different  combinations  of  experimental 
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order  for  the  same  groups  of  subjects.  For  example,  in  an 
experiment  with  three,  levels  of  aiding  (full,  partial,  and 
none),  three  groups  of  subjects  would  participate  in  different 
orders  with  the  levels.  (See  Figure  9).  Thus,  group  A  would 
use  full  aiding  first,  partial  aiding  second,  and  no  aiding 
third.  However,  group  C  would  use  no  aiding  first,  full  aiding 
second,  and  partial  aiding  third,  etc.  In  addition  to  the  Latin 
Square,  other  economical  designs  will  be  used  according  to 
Crolotte,  A.  and  Dechter,  R.,  "Plan  for  TCO  Testing  and  Eval- 
uation'*,  Perceptronics  Report  PFTR-1 082-1 1-79,  November  15,  1979. 
These  designs  include  one-factor  two-level  experiments  such  as 
randomized  repetition,  cross-over  repetition,  and  non-rectangular 
-  Latin  Squares,  etc,  (See  Figure  10) 

Data  analysis  for  these  designs  consists  of  a  one-way  analysis 
of  variance  (ANOVA).  A  summary  of  the  one-way  ANOVA  method 
is  described  in  Figure  11. 
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Latin  Square  Design 
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APPENDIX  A 

DECISION  DIAGRAM  FOR  GAME 

The  following  decision  diagram  shows  the  decision  points, 
event  points,  report  points,  and  identification  points 
for  the  Scenario  Generator  game. 


Flow-d  iagrarr,  for  data  and  decisions  in  the  (i,j)  stage.  The 
tripiet  (F,i,j)  denotes  the  state  of  entering  the  (i,j)  stage 
with  force  F. 

Random  Variables 

-  Own  force  on  entry 

F  -  Own  force  after  an  attack 

Xj  -  InitiaJ  report  regarding  enemy  strength 

*2  -  Report  obtained  by  shallow  reconnaissance 

x^  -  Report  obtained  by  deep  reconnaissance 

Dj  -  Being  detected  during  shallow  reconnai sance 


D_  -  Being  detected  during  deep  reconnaissance 


% 

APPENDIX  B 

DECISION  TABLES  FOR  THE  GAME 


The  following  tables  show  the  optimal  decisions  for  all 
encountered  situations  in  the  Scenario  Generator  game. 
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